1. Field of the Invention
The present invention relates to the field of data storage, and more particularly, to a method and system for compressing and archiving a real-time data-feed in a manner which most efficiently utilizes storage space.
2. Description of the Related Art
The management and storage of log files, files that list actions that have occurred, is imperative in the information age. Log files are used in many different areas. In the telecommunications industry, log files are used to record call activity for each subscriber. In e-commerce, log files are maintained by web servers that list every request made to the server to keep track of where visitors are coming from, how often they return, and how they navigate through a site. This information is necessary for many purposes, including billing, consumer profiling, and marketing.
As one may imagine, these log files are becoming increasingly larger, especially log files detailing daily call activity in a telecommunications network. If these files become too large, they will exceed the size of the storage medium on which they are to be archived. While the file in these instances may be split in half, with each half stored on a separate storage medium, this process is not the most efficient method concerning the space of the storage medium, for each half may leave room on the storage medium that could be used to store other files.
Another problem with current archiving procedures involves the response to system faults. Currently, if there is a fault, crash, or other disruption in the activity of the system, the archiving process is interrupted. To resume the archiving process, typically a human user must evaluate the progress of the archiving before the system fault, and must take actions to resume the archiving process. This task may involve manually zipping data files, if the system was “in the middle” of zipping data files, which may take hours or even days of tedious work by a human user to complete.
Therefore, there is a need in the art for archiving files in such a manner as to maximize the space of the storage medium storing the files. There is also a need in the art for a method and system for archiving data files onto a storage medium when the data file is larger than the size of the storage medium. Furthermore, there is a need in the art for providing a self-recovering mechanism in such a file archiving system to continue the archiving process after a system fault without substantial human interaction.